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REMARKS — General 

By the above amendment, Applicants have amended the claims to define the invention more 
particularly and distinctly so as to overcome the technical rejections and define the invention 
patentably over the prior art. 

1 . In response to paragraph 6 in the last O.A., applicants canceled dependent claims 20, 25, 
30, and 35, and amended independent claims 17 and 27 by adding limitations of "measuring 
confidences about the presence of the target object based on a combination of color and motion 
cues in each of the images captured by said imaging sensors", "the color and motion cues are 
spatially spread out using averaging before they are combined", and "initially distributing the 3D 

r 

target location hypotheses randomly in spaces viewed by said imaging sensors". Applicants 
believe the independent claims are allowable with the reasons discussed in [pages 1 1-19 of the 
remarks in applicants' last amendment, (Amendment B)]. Accordingly, applicants respectfully 
request reconsideration of the currently amended claims 17 and 27. 

2. In response to paragraph 9 in the last O.A., applicants canceled claims 26 and 36. 
Applicants added the limitations, "d) calculating locations of new hypotheses by adding offsets 
to locations of old hypotheses, wherein the offsets are randomly drawn between two numbers," 
and "d) means for calculating locations of new hypotheses by adding offsets to locations of old 
hypotheses, wherein the offsets are randomly drawn between two numbers," to currently 
amended claims 17 and 27, respectively. 
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Applicants disclosed the limitation in the specification filed on 7/12/2003, (Applicants, 
page 17, lines 8-13, and page 19, lines 11-14). 

Applicants believe the limitation is clearly supported by the specification and more 
particularly define the invention in a patentable manner, with the reasons discussed in [page 25, 
line 16 - page 26, line 9 in applicants' last amendment, (Amendment B)]. Accordingly, 
applicants respectfully request reconsideration of the objection, as now applicable to the 
currently amended claims 17 and 27. 

3. In response to paragraph 10 in the last O.A., applicants amended claims 17 and 27, so 
that the currently amended claims do not have the phrase "for combining the measurements". 

The Rejection Of The Claims Under § 112 

4. In response to paragraphs 1 1-30 in the last O.A., applicants amended claims as follows. 
Accordingly, applicants respectfully request reconsideration of this rejection. 

5. In response to paragraph 12 in the last O.A., applicants canceled claims 26 and 36, and 
applicants added the limitations in the canceled claims to claims 17 and 27, respectively, as 
explained above in paragraph 2. The limitations in the claims, as now applicable to the currently 
amended claims 17 and 27, are supported in the specification filed on 7/12/2003, (Applicants, 
page 17, lines 8-13, and page 19, lines 11-14). Accordingly, applicants respectfully request 
reconsideration of the rejection with respect to claims 17 and 27. 
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6. In response to paragraph 15 in the last O.A., applicants amended claims 17 and 27, so 
that the currently amended claims do not have the phrase "examples. . .can", respectively. 

7. In response to paragraph 16 in the last O.A., the limitation "the final target object 
location" has been replaced with the limitation "the 3D location of the target object" in the 
currently amended claims 17 and 27, respectively. 

8. In response to paragraph 17 in the last O.A., the limitation "the weighted mean" has been 
replaced with the limitation "a weighted mean" in the currently amended claims 17 and 27, 
respectively. 

9. In response to paragraph 18 in the last O.A., the limitation "the measurements" has been 
removed in the currently amended claims 17 and 27, respectively. 

10. In response to paragraph 19 in the last O.A., claims 19 and 29 have been canceled. 

11. In response to paragraph 20 in the last O. A., claims 1 9 and 29 have been canceled. 

12. In response to paragraph 21 in the last O.A., claims 19 and 29 have been canceled. 
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13. In response to paragraph 22 in the last O.A., claims 20 and 30 have been canceled. As 
explained above in paragraph 1, the limitation "a combination of color and. motion cues" has 
been added to the currently amended claims 1 7 and 27 5 respectively. 

14. In response to paragraph 23 in the last O.A., claims 20 and 30 have been canceled. The 
limitation "the edges" has been canceled, respectively. 

15. In response to paragraph 24 in the last O.A., claims 20 and 30 have been canceled. The 
limitation "the interior" has been canceled, respectively. 

16. In response to paragraph 25 in the last O.A., the limitation "of the new hypotheses" has 
been replaced with the limitation "in the weighted mean of all 3D target location hypotheses" in 
the currently amended claims 23 and 33, respectively. 

17. In response to paragraph 26 in the last O.A., the limitation "the weights" has been 
replaced with the limitation "weights" in the currently amended claims 23 and 33, respectively. 
The "weights" are "in the weighted mean of all 3D target location hypotheses" in the currently 
amended claims 23 and 33, respectively. 
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1 8. In response to paragraph 27 in the last O.A. ? claims 25 and 35 have been canceled. As 
explained above in paragraph 1 , the limitation "spaces viewed by said imaging sensors" has been 
added to the currently amended claims 1 7 and 27, respectively. 

19. In response to paragraph 28 in the last O.A., applicants amended the claims to read so 
that all "location hypotheses" that correspond to the "3D target location hypotheses" to read the 
same. Regarding claims 19 and 29, they have been canceled. 

20. In response to paragraph 29 in the last O.A., the limitation "step of d) projecting each of 
the 3D target location hypotheses" has been added to claim 18, the limitation "step of b) 
maintaining a large number of 3D target location hypotheses" has been added to claim 23, the 
limitation "d) means for projecting each of the 3D target location hypotheses" has been added to 
claim 28, and the limitation "b) means for maintaining a large number of 3D target location 
hypotheses" has been added to claim 33 to clarify these steps and means correspond to the steps 
and means claimed in the corresponding independent claims, respectively. Regarding claims 1 9- 
20 and 29-30, they have been canceled. 

21 . In response to paragraph 30 in the last O.A., the limitation "maintaining 3D target 
location hypothesis" has been removed in the currently amended claims 23 and 33, respectively. 

Regarding claims 24 and 34, they have been canceled. 
Regarding claims 26 and 36, they have been canceled. 
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The Rejection Of The Claims Under § 103 

22. The last O.A. rejected claims 17-36 under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent No. 6999624 (Nelson) in view of U.S. Patent Application Publication No. 
20010008561 (Paul et al, hereinafter Paul), and further in view of U.S. Patent No. 5323470 (Kara 
et al, hereinafter Kara). 

Applicants amended the claims as follows: 

The Rejection Of Claim 27 on Nelson, Paul, and Kara Overcome 

i 

23. Applicants request reconsideration of the rejection, as now applicable to the currently 
amended claim 27, for the following at least five reasons: 

(1) Target object tracking based on novel usage of multiple 3D target location hypotheses is 
clearly foreign to Nelson, Paul, and Kara. The novel step in the present invention was not 
suggested in the reference or any prior art, produces new and unexpected results, and hence is 
unobvious and patentable over the reference. 

(2) Random distribution of the hypotheses is foreign to Nelson, Paul, and Kara, and the focus of 
the idea in Paul with regard to the initialization of target object location is contradictory to that of 
the present invention. 

(3) Modeling arbitrary shape and contour of the target object is foreign to Nelson, Paul, and Kara. 

(4) There is no justification in Nelson, Paul, and Kara, or in any other prior art separate from 
applicants' disclosure, which suggests that these references be combined, much less be combined 
in the manner proposed. 
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(5) Since the fundamental approaches in applicants 5 system are clearly foreign to Nelson, Paul, 
and Kara, even if Nelson, Paul, and Kara were to be combined in the manner proposed in the last 
O.A., the proposed combination would not show all the novel features of claim 27. 

The novel and unobvious approaches in applicants' system are clearly foreign to Nelson, 
Paul, and Kara, and they were not suggested in the reference or any prior art. Applicants 
respectfully note that the significant differences in the approaches between applicants' present 
invention and Nelson, Paul, and Kara are not to be underestimated or misunderstood. 

Target Object Tracking Based On Novel Usage Of Multiple 3D Target Location 
Hypotheses Is Clearly Foreign To Nelson, Paul, and Kara 

24. Applicants noted, "Projective geometry states, that we can obtain from the projected 
locations tl and r R of a point R, the corresponding 3D coordinate through triangulation. Most 
stereo algorithms rely on this principle by finding correspondences between two images and 
recovering depth through triangulation. However, finding correspondences between views is 
challenging," (Applicants, page 10, lines 3-7). 

As a solution to the challenge, applicants disclosed a novel approach to the problem with 
the measurement of the confidence in the hypothesis as in "given a hypothesized hand location in 
world coordinates R, the image data is used to measure the confidence in this hypothesis, i.e., the 
degree to which the image data supports the hypothesis that a hand is indeed present at the 2D 
locations corresponding to the world coordinate R" (Applicants, page 10, lines 11-14). 

Especially, to increase the accuracy of the approach, applicants disclosed, "a large 
number of such hypotheses are maintained and combined to get a statistical measure of the true 
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hand location in world coordinate space" (Applicants, page 10, lines 14-16). "In the illustration 
FIG. 1 1, the value is N=14 but in practice, this number is much larger." (Applicants, page 16, 
lines 7-8). Typically, in an exemplary embodiment of the present invention, the number of 
hypotheses could be between 40 and 500, but depending on the parameters, such as the size of a 
hand, the number could be more or less. 

The usage of "a large number of such hypotheses" is clearly foreign to Nelson, Paul, and 

Kara. 

As the last O.A. also noted, Nelson did not explicitly disclose the usage of "a large 
number of 3D location hypotheses". Nelson's "overlapping windowed portions" in (Nelson, 
column 3, lines 42-46) is clearly different from the "3D target location hypotheses" in 
applicants' present invention. 

In addition, Nelson explicitly disclosed the grouping process of the windows in (Nelson, 
column 6, lines 18-28), which is clearly irrelevant to the "3D target location hypotheses" in 
applicants' present invention. 

Furthermore, Nelson explicitly disclosed the group score calculation process in "with the 
groups of possible target windows being formed, the method of the present invention next 
calculates a group score at step 40 for each group. Each group score is based on the maximum of 
one or several classification scores for each of the possible target windows assigned to that 
group." (Nelson, column 6, lines 29-33). Nelson further discussed some examples of the group 
score calculation in (Nelson, column 6, lines 34-52). 
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This clearly shows Nelson's approach in the window grouping process and group score 
calculation are clearly foreign to the "3D target location hypotheses" in applicants' present 
invention. 

Paul explicitly disclosed, "The system tracks objects based on the color, motion and 
shape of the object in the image. The tracking algorithm uses a unique color matching technique 
which uses minimal computation. This color matching function is used to compute three 
measures of the target's probable location based on the target color, shape and motion. It then 
computes the most probable location of the target using a weighting technique." (Paul, page 1, 
paragraph 10). 

Paul clearly noted that their system "tracks objects based on the color, motion and shape 
of the object in the image" using "a unique color matching technique". Applicants' present 
invention uses only color and motion cues. Regarding the "shape", as discussed in (Applicants' 
amendment B, page 15, line 1 - page 17, line 3), applicants' present invention tries to avoid 
using the shape-based approach for target object tracking. 

Furthermore, Paul clearly noted, "It then computes the most probable location of the 
target using a weighting technique." Paul did not disclose the novel usage of "a large number of 
3D location hypotheses" in relation to the "probable location" in anywhere in their disclosure, 
and Paul is entirely foreign to the concept. 

Especially, in the description for the "Color Matching" (Paul, page 2, paragraph 24 - 
page 3, paragraph 31), Paul explicitly disclosed, "The two thresholds will decide the shape of the 
matching color cone. A threshold on the angle of the color cone and another threshold on the 
minimum length of the color vector decides the matching color space. Thus, any pixel whose 
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color lies within the truncated cone in color space will be considered as having the same color as 
the target." (Paul, page 2, paragraph 26) and "ColorMatch (R,G,B)={dmdaif((dml<d 
m<dmh)&(dl l<da<dah))0 otherwise" (Paul, page 2, paragraph 27), which is clearly 
different from the confidence measure (values between zero and one) that is given by the 
function 1.1 in (Applicants, page 1 1, line 17). 

* 

In Paul's description for the motion detection (Paul, page 3, paragraphs 35-40), Paul also 
did not explicitly disclose the novel usage of "a large number of 3D location hypotheses" 

Neither Nelson nor Paul disclose the novel usage of "a large number of 3D location 
hypotheses" (please also refer to Applicants, page 16- page 17 and Fig. 1 1 and 12) in anywhere 
in their disclosure, and they are clearly foreign to the concept. Kara is foreign to the idea. 
Therefore, there is no logical explanation that the combination of the three prior arts can create 
the same result in the manner applicants' present invention proposed. 

Random Distribution Of The Hypotheses Is Foreign To Nelson, Paul, and Kara 

25. Applicants disclosed random distribution of the hypotheses at the starting point of the 
hand tracking, "The target location hypotheses H i are generated over time as follows: When the 

method starts (say at time t=0), and no information from the past is available, the R\ at time t=0 

are randomly distributed in the interaction volume 400. For all later times t>0, N new hypotheses 
are created from the N old hypotheses available from the previous time step t-dt as follows: For 
the creation of the i-th (i between 1 and N) hypothesis, a number j between 1 and N is randomly 

drawn such that the probability of drawing a number n is equal to the probability W"_ } ." 

(Applicants, page 17, lines 2-8). This approach is clearly foreign to Nelson, Paul, and Kara. 
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Furthermore, Paul explicitly suggested the usage of a user's manual input for the 
initialization step, as shown "It then allows the user to select the target manually using the mouse 
or automatically by moving the target to a predetermined position in the scene. At the point of 
initialization, the color, the shape and location of the target are computed and stored." (Paul, 
page 2, paragraph 20, please also refer to Paul, page 1, paragraph 11, and Paul, page 2, paragraph 
18), which the present invention identifies as a problem in an automatic object tracking system 
and the novel approach of the present invention overcomes. 

Applicants' preliminary preparation step, such as the calibration of the cameras, uses 
manual input for determining the image locations of the calibration point. 

However, the tracking of the target object in applicants' system is processed completely 
automatically without requiring any manual initialization. 

This further shows the approach and focus of the idea in Paul is contradictory to that of 
the present invention. 

Furthermore, claim 27 further adds limitation of means for calculating locations of new 
hypotheses by adding offsets to locations of old hypotheses, wherein the offsets are randomly 
drawn between two numbers. 

Applicants explicitly disclosed, "For all later times t>0, N new hypotheses are created 
from the N old hypotheses available from the previous time step t-dt as follows: For the creation 
of the Uth (i between 1 and N) hypothesis, a number j between 1 and N is randomly drawn such 

that the probability of drawing a number n is equal to the probability W t % . The location of the 

new i-th hypothesis is given by the location of the old hypothesis plus a small offset 

AR[ = (AX' t 9 AY/ 9 AZ,' ) , where each of the three offsets AX[ , AY/ , AZ\ are randomly drawn 
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between two numbers -OFFSET and +OFFSET (- 5mm and +5mm for the preferred 

embodiment described here). The weight of the new hypotheses with location R l t = R J t _ x + A/?,' is 

given by the confidence function CONFi R (R,t)" (Applicants, page 17 5 lines 5-14). 

Nelson is entirely foreign to the idea of drawing random displacements and adding them 
to the 3D target location hypotheses, especially for tracking a target object, since Nelson is only 
a method for classifying targets in a digital image rather than a method for tracking a target 
object. 

Modeling Arbitrary Shape And Contour Of The Target Object Is Foreign To Nelson, Paul, 
and Kara 

26. As discussed above, applicants' system is based on a novel usage of multiple hypotheses 
for target object tracking, and "In step 1440, N new hypotheses are created from the hypotheses 
at the previous time step t-dt by randomly selecting previous hypotheses with probability equal 
to the hypothesis weights" (Applicants, page 19, lines 11-13). Then, "the locations of the newly 
selected hypotheses are displaced randomly in step 1450." (Applicants, page 19, lines 13-14). 

Therefore, applicants' present invention is not bound by any fixed shape or contour of a 
certain target object model because the locations of the "N new hypotheses" can change 
randomly at every step. This capability of modeling any arbitrary shape and contour for the 
target object is foreign to Nelson, Paul, and Kara. Nelson's approach for the classification of 
targets does not teach a method for tracking target objects. Paul explicitly disclosed a usage of a 
shape matching in (Paul, page 3, paragraph 32-34). Kara teaches, "comparing a pre-stored image 
of the object with the current image from the camera." in (Kara, column 5, lines 12-21). 
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Nelson, Paul, and Kara Do Not Contain Any Justification To Support The Combination, 
Much Less In The Manner Proposed 

27. Due to the differences mentioned above in the fundamental approaches for applying 
computer vision algorithms, there is a significant lack of proof that supports the proposed 
combination in the last O.A. Since Nelson, Paul, and Kara do not show the novel and unobvious 
features in applicants' present invention, there is clearly no logical connection that suggests the 
proposed combination. Furthermore, the prior arts themselves do not contain any suggestion that 
they be combined in the manner proposed. 

Fundamentally, Nelson is a method of classification of targets in a digital image by 
evaluating windowed portions of the image, and Nelson is entirely foreign to the idea of target 
object tracking based on novel usage of multiple target location hypotheses. It is well known that 
the tasks in the classification of a target object in an image are non-trivially different from those 
in the tracking of a target object in the practiced art. There is clearly no justification that Nelson 
can suggest the ideas of tracking a target object utilizing 3D target location hypotheses in the 
manner proposed in the present invention. 

Even If Nelson, Paul, and Kara Were To Be Combined In The Manner Proposed, The 
Proposed Combination Would Not Show AH The Novel Features Of Claim 27 

28. Even if the combination of Nelson, Paul, and Kara were legally justified, claim 27 would 
still have novel and unobvious features over the proposed combination, since the fundamental 
approaches in applicants' present invention are clearly foreign to Nelson, Paul, and Kara. 
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Specifically, clauses (b), (c) 5 (d), (e), (f), and (g) clearly distinguish applicants' present invention 
from Nelson, Paul, and Kara, or any possible combination thereof. None of Nelson, Paul, and 
Kara shows the features in clauses (b), (c), (d), (e), (f), and (g). 

29. In paragraph 33 of the last O.A., it is noted that claim 17 is rejected for the same reasons 
as claim 27, and claim 17 distinguishes from claim 27 only in that claim 17 is a method claim 
and claim 27 is an apparatus claim. 

Applicants amended claim 17, which recites limitations that are similar and in the same 
scope of invention as to those in claim 27 above. 

Therefore, applicants request reconsideration of the amended claim 1 7 for the same 
reasons as stated above in regards to claim 27. 

The Dependent Claims Are a Fortiori Patentable Over Nelson, Paul, and Kara 

30. Dependent claims 28 to 36 incorporate all the subject matter of claim 27 and add 
additional subject matter, which makes them a fortiori and independently patentable over the 
references. 

Applicants amended the claims 28 to 36 as follows: 

31. In response to paragraph 34 in the last O.A., claim 28 further adds means for utilizing 
projections that are obtained by calibrating said imaging sensors with respect to a reference 
coordinate system. 
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Applicants disclosed the usage of an exemplary calibration cube with eight calibration 
points, "as illustrated in Fig. 9, the camera calibration matrices are obtained by placing a 
calibration target such as a cube (1000) with a set of Nc calibration points (e.g., 1010a, 1010b) 
with known world coordinates in the viewing area. In FIG. 9, the calibration cube has Nc=8 
calibration points." (Applicants, page 17, line 16 - page 18, line 17). 

Paul disclosed, "At the point of initialization, the color, the shape and location of the 
target are computed and stored. Once the target is initialized, we compute an estimate of the 
target location using target dynamics. We then compute the actual location using the color, shape 
and motion information with respect to a region centered at the estimated location." (Paul, page 2, 
paragraph 20). The initialization is a clearly different process from the calibration process, and 
the step of projecting a large number of target location hypotheses based on the calibration 
process in the manner proposed by the present invention is not shown in Paul. 

32. In response to paragraph 35 in the last O.A., claim 29 has been canceled. 

33. In response to paragraph 36 in the last O.A., claim 30 has been canceled. 

34. In response to paragraph 37 in the last O.A., claim 3 1 further adds means for calculating 
the color cues using a color model of the target object, wherein the color model of the target 
object is represented by a histogram that is estimated by collecting color samples of the target 
object. 
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The novel usage of color for calculating the color cues of the confidence measure as 
disclosed in (Applicants, page 11, line 1 - page 12, line 2), including the function 1.1 in 
(Applicants, page 1 1, line 17), is clearly foreign to Paul and Nelson. Neither Paul nor Nelson 
discloses how the color cues are used to measure confidences for the 3D target location 
hypotheses in the manner proposed in the present invention. 

35. In response to paragraph 38 in the last O.A., claim 32 further adds means for calculating 
motion cues by measuring differences between images captured sequentially by the imaging 
sensors. 

The novel usage of motion for calculating the motion cues of the confidence measure as 
disclosed in (Applicants, page 12, lines 3 - 20), including the function 1.2 in (Applicants, page 
12, line 17), is clearly foreign to Paul and Nelson. Neither Paul nor Nelson discloses how the 
motion cues are used to measure confidences for the 3D target location hypotheses in the manner 
proposed in the present invention. 

36. In response to paragraph 39 in the last O.A., claim 33 further adds means for creating a 
set of 3D target location hypotheses at each time step. The target location hypothesis 
maintenance disclosed in the manner proposed by the applicants' system (Applicants, page 17, 
lines 1-14) is entirely foreign to Paul. 

Paul explicitly disclosed, "If motion has occurred, there will be sufficient change in the 
intensities in the region. The motion detection function will trigger if a sufficient number of 
pixels change intensity by a certain threshold value. This detection phase eliminates unnecessary 
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computation when the object is stationary." (Paul, page 3, paragraph 36), which is clearly 
different from the idea of creating a set of 3D target location hypotheses at each time step in "For 
all later times t>0, N new hypotheses are created from the N old hypotheses available from the 
previous time step t-dt as follows:" (Applicants, page 17, lines 5-7). 

Furthermore, the novel step of creating a set of 3D target location hypotheses based on 
the confidence function CONFLR(R,t) (Applicants, page 17, lines 13-14, please also refer to 
function 1.3 in Applicants, page 14, lines 1 1-21 and function 1.4 in Applicants, page 15, lines 
10-16) at each time step is entirely foreign to Paul. Paul only very briefly mentioned the usage of 
tracking for a target in 3D in "Another uses the tracking in two stereo images to track the target 
in 3D." (Paul, page 4, paragraph 45). This clearly shows that Paul is foreign to the idea of 
creating a set of 3D target location hypotheses based on the confidence function CONFLR(R,t). 

37. In response to paragraph 40 in the last O.A., claim 34 has been canceled. 

38. In response to paragraph 41 in the last O.A., claim 35 has been canceled. 

39. In response to paragraph 42 in the last O.A., claim 36 has been canceled. 



40. Accordingly, applicants submit that the dependent claims are a fortiori patentable and 
should also be allowed. 
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41 . In paragraph 43 of the last O.A., it is noted that claims 1 8-26 are rejected for the same 
reasons as claims 28-36, and claims 18-26 distinguish from claims 28-36 only in that they have 
different dependencies. 

Applicants amended claims 18-26, which recite limitations that are similar and in the 
same scope of invention as to those in claims 28-36 above, respectively. 

Therefore, applicants request reconsideration of the amended claims 1 8-26 for the same 
reasons as stated above in regards to claims 28-36, respectively. 
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CONCLUSION 

For all the above reasons, applicants submit that the specification and claims are now in proper 
form, and that the claims all define patentably over the prior art. Therefore they submit that this 
application is in condition for allowance now, which action they respectfully solicit. 

Conditional Request for Constructive Assistance 

Applicants have amended the specification and claims of this application so that they are proper, 
definite, and define novel structure, which is also unobvious. If, for any reason this application is 
not believed to be in full condition for allowance, applicants very respectfully request the 
constructive assistance and suggestions of the Examiner pursuant to M.P.E.P. § 2173.02 and § 
707.07(j) in order that the undersigned can place this application in allowable condition. 

Very respectfully, 




Rajeev Sharma 

- Applicants Pro Se 



403 South Allen Street, Suite 101 

State College, PA 16801 

Tel. (814) 867-8977; Fax (814) 867-8957 
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